An Optimized Web Feed Aggregation Approach for Generic Feed Types

Web feeds are a popular way to access updates for content in the World Wide Web. Unfortunately, the technology behind web feeds is based on polling. Thus, clients ask the feed server regularly for updates. There are two concurrent problems with this approach. First, many times a client asks for updates, there is no new item and second, if the client’s update interval is too large it might be notified too late or even miss items. In this work we present adaptive feed polling algorithms. The algorithms learn from the previous behaviors of feeds and predict their future behaviors. To evaluate these algorithms we created a real set of over 180,000 diversified feeds and collected a dataset of their updates for a time of three weeks. We tested our adaptive algorithms on this set and show that adaptive feed polling reduces traffic significantly and provides near-real-time updates.